Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add moz-webgpu-cts process-reports subcmd. with multiple --presets #14

Merged
merged 6 commits into from
Nov 6, 2023

Conversation

@ErichDonGubler ErichDonGubler self-assigned this Oct 19, 2023
@ErichDonGubler ErichDonGubler force-pushed the adjust-exps branch 3 times, most recently from cd7902c to 454a4db Compare October 27, 2023 03:18
@ErichDonGubler
Copy link
Collaborator Author

ErichDonGubler commented Oct 27, 2023

I was struggling to get traction on resolving conflicts between expected outcomes in metadata and reported outcomes. I realized I was missing an answer to the question, “How do I structure new expectations' conditional values once reports have been analyzed?” An example answer: one could organize by platform:

[test]
  [subtest]
    expected:
      if os == "win": FAIL
      if os == "linux": TIMEOUT

…or one could organize by build profile:

[test]
  [subtest]
    expected:
      if debug: FAIL
      if not debug: TIMEOUT

…or even potentially mix them, which seems reasonable, but only muddies up the solution space for “what do we emit”:

[test]
  [subtest]
    expected:
      if os == "win": NOTRUN
      if debug: FAIL

In addition to the variables explored above, WPT upstream lets users define what I'll call “branching points” in expectation structure for any run info variable. WPT upstream's update-expectations script actually has a full-blown configuration schema for defining the these branching points. These branching points can still “collapse” into a single outcome if all variations of a branching point are the same. This is neat, and solves the question we ask above. However, this leaves another problem: to implement this, we'd have to make some gnarly, complicated, dynamic code.1

I determined to answer the question I've posed instead by (1) baking a branch point configuration into code for now, and (2) granting moz-webgpu-cts a limited knowledge of all platforms we intend to run on for now. expected properties will be divided first by supported platform, then by build profile, eliding conditionals if all variants are the same. In practice, this may mean we (1) lose granularity with things like Firefox release (e.g., nightly vs. stable) and (2) we may duplicate the same expected outcomes between different platforms on the same build profile. I think these are pretty acceptable trade-offs for getting something tractable now. We can update this model later, if this MVP needs tweaking down the road.

Footnotes

  1. I wouldn't be surprised if the dynamism hurts its ceiling for performance, too.

@ErichDonGubler ErichDonGubler force-pushed the adjust-exps branch 3 times, most recently from 4d39480 to eb6f649 Compare October 27, 2023 14:26
@ErichDonGubler ErichDonGubler changed the base branch from main to fixups October 27, 2023 14:26
@ErichDonGubler ErichDonGubler deleted the branch main October 27, 2023 14:32
@ErichDonGubler ErichDonGubler changed the base branch from fixups to main October 27, 2023 16:08
@ErichDonGubler ErichDonGubler force-pushed the adjust-exps branch 14 times, most recently from 20f8b91 to 9a92a59 Compare November 1, 2023 21:30
@ErichDonGubler ErichDonGubler force-pushed the adjust-exps branch 2 times, most recently from cfa3114 to d35ca95 Compare November 2, 2023 03:38
@ErichDonGubler ErichDonGubler changed the title WIP: feat(cli): add moz-webgpu-cts adjust-exps subcmd. WIP: feat(cli): add moz-webgpu-cts process-reports subcmd. Nov 2, 2023
@ErichDonGubler ErichDonGubler force-pushed the adjust-exps branch 2 times, most recently from 323f8a3 to 02ea3ee Compare November 3, 2023 18:10
@ErichDonGubler
Copy link
Collaborator Author

ErichDonGubler commented Nov 6, 2023

Removed TestExecutionResult::duration with dd3cad3 for now; broke out #53 for later work.

@ErichDonGubler ErichDonGubler marked this pull request as ready for review November 6, 2023 17:08
@ErichDonGubler ErichDonGubler changed the title WIP: feat(cli): add moz-webgpu-cts process-reports subcmd. Add moz-webgpu-cts process-reports subcmd. with multiple --presets Nov 6, 2023
@ErichDonGubler ErichDonGubler merged commit 7d4e299 into main Nov 6, 2023
21 checks passed
@ErichDonGubler ErichDonGubler deleted the adjust-exps branch November 6, 2023 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Introduce moz-webgpu-cts process-reports
1 participant